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THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
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- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
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Detailed Action 
Preface 

1. The first and second equations listed in column 4 of Kirk (U.S. Patent 4803548) will be referred 
to, respectively, as eq.l and eq.2 in this document. 

2. Luminance will be referred to interchangeably with the green color channel in the remainder of 
this document. Chrominance will be taken to mean a red or blue color channel. This is consistent with 
Adams' convention (see page 146, section 3.4 of "Interactions Between Color Plane Interpolation and 
Other Image Processing Functions in Electronic Photography") 



Drawings 

3. The proposed drawing correction and/or the proposed substitute sheets of drawings, filed on 
January 10, 2001 have been approved. A proper drawing correction or corrected drawings are required in 
reply to the Office action to avoid abandonment of the application. The correction to the drawings will not 
be held in abeyance. 

4. The Patent and Trademark Office no longer makes drawing changes. See 1017 O.G. 4. It is 
applicant's responsibility to ensure that the drawings are corrected. Corrections must be made in 
accordance with the instructions below. 

INFORMATION ON HOW TO EFFECT DRAWING CHANGES 
Replacement Drawing Sheets. 

5. Drawing changes must be made by presenting replacement figures which incorporate the desired 
changes and which comply with 37 CFR 1.84. An explanation of the changes made must be presented 
either in the drawing amendments, or remarks, section of the amendment. Any replacement drawing sheet 
must be identified in the top margin as "Replacement Sheet" and include all of the figures appearing on the 
immediate prior version of the sheet, even though only one figure may be amended. The figure or figure 
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number of the amended drawing(s) must not be labeled as "amended." If the changes to the drawing 
figure(s) are not accepted by the examiner, applicant will be notified of any required corrective action in 
the next Office action. No further drawing submission will be required, unless applicant is notified. 

6. Identifying indicia, if provided, should include the title of the invention, inventor's name, and 
application number, or docket number (if any) if an application number has not been assigned to the 
application. If this information is provided, it must be placed on the front of each sheet and centered within 
the top margin. 

Annotated Drawing Sheets. 

7. A marked-up copy of any amended drawing figure, including annotations indicating the changes 
made, may be submitted or required by the examiner. The annotated drawing sheets must be clearly 
labeled as "Annotated Marked-up Drawings" and accompany the replacement sheets. 

Timing of Corrections. 

8. Applicant is required to submit acceptable corrected drawings within the time period set in the 
Office action. See 37 CFR 1 .85(a). Failure to take corrective action within the set period will result in 
ABANDONMENT of the application. 

9. If corrected drawings are required in a Notice of Allowability (PTOL-37), the new drawings 
MUST be filed within the THREE MONTH shortened statutory period set for reply in the "Notice of 
Allowability." Extensions of time may NOT be obtained under the provisions of 37 CFR 1 . 136 for filing 
the corrected drawings after the mailing of a Notice of Allowability. 



Claims 

Objections 

10. Claims 4, 6, and 14 are objected to because of the following informalities. This passage was taken 
from claim 4: 

.. .computes the coefficients ac and b'c using the following equations, in which \xc is the mean and o^c is the variance of 
green and red color channels C, respectively... 
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Phrased as such, it can be interpreted from claim 4 that the green color channel and the red channel both 
have a mean of u c and a variance of a C) though it is clear from other portions of the applicant's disclosure 
that |i c and ct c are the mean and variance, respectively, of a color channel C, where C = R, G, or B, 
corresponding to red, green, or blue color channels, respectively. The correspondence between the red, 
green, and blue color channels and the subscripts of the various variables and parameters found in the 
subsequent equations of claim 4 should be explicitly defined. Claim 14 is similarly objected to. Appropriate 
correction is required. 

1 1 . Claim 6 recites the following: 

The apparatus according to claim 5, wherein the set sums of color channels comprises a pair of sets of sums of 
color channels. 

It is clear from the preceding claims that the applicant is referring to the set of sums of color channels as 
opposed to the set sums of color channel. The potential for the misinterpretation of claim 6 should be 
obvious. Claim 16 is similarly objected to. Appropriate correction is required. 

Rejections Under U.S.C. 8 1 12(2) 

12. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

13. Claims 4 and 14 are rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for failing 
to particularly point out and distinctly claim the subject matter which applicant regards as the invention. 
There is no mention of a blue color channel in claims 4 and 14. Specifically, it is not suggested in claims 4 
and 14 that C can correspond also to a blue color channel. It is, therefore, unclear as to what color channel 
the variables and parameters, a b) b' b , a B , and u B correspond to. The computation of the regression 
coefficients a and b' for the blue color channel - a computation essential to the applicant's claimed 
invention - is therefore absent from the claims. 



Rejections Under U.S.C. § 102(b) 

14. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for 
the rejections under this section made in this Office action: 
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A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in 
this country, more than one year prior to the date of application for patent in the United States. 

15. Claims 1-2 and 1 1-12 are rejected under 35 U.S.C. 102(b) as being anticipated by Adams 
("Interactions Between Color plane Interpolation and Other Image Processing Functions in Electronic 
Photography"). 

16. The following is in regard to claim I. As discussed by Adams, electronic cameras using a single 
CCD detector, employing a color filter array (CFA), capture (i.e. samples) three color planes of color 
information. Such systems (e.g. single sensor Bayer CFA system - see Adams page 144, Section 1.2) 
produce images such that each image pixel contains information about only one of three spectral bands. 
These images are subsequently interpolated to reconstruct the image to have three full-resolution color 
planes. See page 144 of Adams. In this regard, such electronic cameras provide an "apparatus for 
reconstructing color filter array images, comprising: an image recording module for generating an image of 
sampled values output from a color filter array; each pixel in the image having a location at which a 
sampled value of one of a plurality of color channels is recorded", in accordance with claim 1. 

17. Adams discusses several interpolation techniques used in the reconstruction of the three full- 
resolution color planes. See, for example, section 3.4 and figure 9 on pages 146-147. Notice that 
interpolation equations (9)-(l 1) provide a set of linear transformations from the sampled value at the 
location of a selected pixel in the image to an estimate of a color channel not recorded by the single-sensor 
CFA system (i.e. the image recoding module) at the location of the selected pixel in the image. Consider, 
for instance, equation (9). Equation (9) can be shown to satisfy the mathematical definition of a linear 
transformation from the sampled value, G2, to the estimate of the color channel not recorded, i.e. B2. 
Clearly, this is in accordance with the "image reconstruction module" of claim 1. Thus, Adams teaches all 
aspects of claim 1 . 

18. The following is in regard to claim 2. Again, referring to equations (9)-(10) of Adams, notice that 
the coefficients of the aforementioned linear transformations are computed. For example, equation (9) can 
be expressed as B2 = cG2, where c=!/ 2 (Bl/Gl + B3/G3) can be considered the coefficient of the linear 
transformation from G2 to B2. In this regard, Adams teaches all aspects of claim 2. 
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19. The following is in regard to claims 11 and 12. Note that claims 1 1 and 12 merely claim the 
method that is implemented by the apparatus of claims 1 and 2, respectively. Therefore, with regard to 
claims 1 1 and 12, arguments analogous to those presented for 1 and 2 are applicable. See the appropriate 
paragraphs above. 



Rejections Under U.S.C. § 103(a) 

20. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of 
this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject 
matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art 
to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made. 

21. Claims 3, 5-10, 13, and 15-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Adams, in view of Kirk (U.S. Patent 4803548). 

22. The following is in regard to claim 3. As shown above, Adams discloses an apparatus in 
accordance with claims 1 and 2. Adams does not, however, disclose an apparatus in accordance with claim 
2, wherein the coefficients of the linear transformation are computed by computing statistics that depend on 
the sampled value but not the position of the samples of at least two color channels within a window of the 
image. 

23. Kirk discloses a method and apparatus of enhancing a video image defined by luminance and 
chrominance information, where for a region of predetermined size (i.e. a window), selected among the 
multitude of such regions composing the video image, the following is done: 

i. determining a statistical relationship between the luminance information and the 
chromaticity information for the selected region; and, 

ii. generating modified chromaticity information for at least a portion of the selected region 
by applying the determined statistical relationship to the luminance information for the 
said portion. 

See Kirk, "Summary of Invention", paragraphs 1-3. Preferably, (i) comprises carrying out a regression 
analysis to generate a relationship between the luminance (L) and a chrominance coordinate (I) of the form, 
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I = C 0 + 2C n F„(L) + where F n is a general function series, such as a power series of L. The intention of 

this analysis is to determine values for the constants C 0 , C b etc. Preferably, a linear relationship (see the 
equations in column 4 of Kirk) is determined between luminance and chromaticity leading to the generation 
of just two regression coefficients for each chromatic signal. See Kirk, column 2, lines 25-39 and column 4, 
lines 4-45. It is clear that the above relationship between L and I is dependent on the values of the 
chromaticity I and luminance L and not the spatial location of the corresponding pixel within the given 
image. It should thus be evident from the preceding discussion that Kirk teaches the computation of the 
linear transformation coefficients in a manner similar, if not identical, to that of claim 3. 

24. Clearly, using regression analysis, in the manner proposed by Kirk, simply provides an alternative 
mathematical device to the interpolation technique, discussed above, for the derivation of a linear 
relationship and/or transformation from one color channel to another. The applicability of Kirk's method 
and apparatus to the reconstruction of CFA images stems from the fact that it provides a means to generate 
a high or full-resolution set of estimated color information (e.g. a full-resolution color plane) from a low 
resolution or incomplete set of color information (e.g. the incomplete color planes of the CFA image). It 
would, therefore, be straightforward for one of ordinary skill in the art to replace the interpolation scheme 
taught by Adams with the regression analysis of Kirk's invention. 

25. In general, the effectiveness of an interpolation technique is a function of the spatial resolution of 
the set of known points. The effectiveness of interpolation, therefore, degrades when sets of color 
information (e.g. a sampled color plane) are sparse. Thus, the quality or effectiveness of a CFA image 
reconstruction method or apparatus becomes directly related to the spatial resolution of the CFA image, 
which, in turn, is related to the density of costly CCD sensors. This is suggested in the "Description of the 
Prior Art" of Kirk. Clearly, a method or apparatus that provided a means for deriving an estimated relation 
between color channels, whereby a reconstructed CFA image can be obtained, and is further independent of 
the spatial resolution of the sampled CFA image, is desirable. Given that the relation derived by the method 
and apparatus of Kirk is not spatially dependent, one would expect that it would be less prone to the 
degradation exhibited by interpolation techniques, and hence less sensitive to the spatial resolution of the 
sampled CFA image. Given this advantage over interpolation techniques of Adams and the apparent 
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interchangeability of those techniques and the regression analysis of Kirk, it would have been obvious to 
one of ordinary skill in the art, at the time of the applicant's claimed invention, to replace the interpolation 
techniques taught by Adams with the regression analysis and items (i) and (ii) above, to perform CFA 
image reconstruction in a manner similar to claim 3. 

26. The following is in regard to claim 5. The regression analysis shown in Kirk defines a linear 
relation between the sets of luminance averages ( L ) and sets of chromaticity averages, 7 and Q , 
respectively. These relations are given by the equations in column 4 of Kirk. See also Kirk, column 3, lines 
15-30, column 4, lines 4-45, and Fig. 2. It can be seen from these equations that a line is fit to the set of L 
and the set of 7 , in eq.l, and a line is fit to the set of L and the set of Q , in eq.2. The regression analysis 
of Kirk (i.e. the line fitting) attempts to derive the regression coefficients, C 0 i, C u , C 0Q , and C 1Q of the said 
equations. Each element of the aforementioned sets of averages corresponds to one of the multitude of 4x4 
regions composing the image. These regions are analogous to the windows of claim 5. Furthermore, since 
averages are weighted sums, the aforementioned sets of averages can each be reasonably considered sets of 
sums of color channels. In computing the averages, such as L , 7 , and Q for each of the said regions, 
color channel information (e.g. L, /, or Q) is typically accumulated sequentially along lines through the 
region (e.g. along rows or along columns). Therefore, the image reconstruction module, obtained by 
combining the teachings of Adams with those of Kirk, in the manner described above, computes the 
regression coefficients, C 0 i and Cu, by fitting a line to a set of weighted sums along lines through a window 
(i.e. the set of L ) and another set of weighted sums along lines through the same window (i.e. the set 

of 7); and computes the regression coefficients, C 0Q , and Cjq, by fitting a line to a set of weighted sums 
along lines through a window (i.e. the set of L ) and another set of weighted sums along lines through the 
same window (i.e. the set of Q ). This is in accordance with claim 5. 

27. The following is in regard to claim 6. Observe from the preceding discussion that, in the 
computation of, say, C 0 i and Cu, the union of the set of I and the set of 7 form a set of sums of color 
channels comprising a pair of sets of sums of color channels. Therefore, the teachings of Adams and Kirk, 
when combined in the manner described above, address all aspects of claim 6. 
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28. The following is in regard to claim 7. Kirk suggests using the least squares fitting technique to fit 
lines (defined by eq.l and eq.2) to the respective chromaticity values and the luminance values. See Kirk, 
column 3, lines 22-29. In this manner, the teachings of Adams and Kirk, when combined as described 
above, address the limitations of claim 7. 



defined by eq. 1 and eq.2, a detennination is made as to whether the said functions fit the corresponding L, I 
data well for the region. Kirk achieves this by obtaining the sum of the squares of the differences between 
each data point and the respective function and comparing with a predetermined threshold. See Kirk 
column 4, lines 32-44. This addresses the claimed subject matter of claim 8 in the following way. First, the 
sum of the squares of the differences (for the sake of brevity, this sum will be referred to here as Z D ), just 
mentioned, provides a quantitative measure of the goodness-of-fit of the derived linear functions with the 
corresponding L or I data of the given region, or conversely, a quantitative measure of the goodness-of-fit 
of the L or I data of the given region with the corresponding linear function. I D can be considered a 
confidence measure, in the same vein as that of the applicant's disclosure, because it and the applicant's 
confidence measure (e.g. Q on page 8 of the applicant's disclosure) both provide a goodness-of-fit measure 
which is later used to restrict outlying or potentially outlying color channel data. Furthermore, Kirk only 
constrains the regions to be of the same size and square or rectangular (Kirk, column 3, lines 14-15). Thus, 
the regions may be chosen such that each forms a row or column. In such a case, S D would be calculated 
per row or column. Lastly, Z D for a given row or column is directly related to the sum of the color channels 
for that row or column. Intuitively, I D has the form, ^ (% { - Xx ) 2 > where %i is a color channel datum (I or 
Q) along the given row or column and Xi * s th e corresponding color channel datum estimated by the 
derived regression model. It can be shown by expanding this expression that Z D is further related to the 
squared difference, (^^ Xi ~ X/Ii %i ^ wnere > £ Xi > * s me sum °f tne given row or column of color 

channel data and Z x i , is the sum of the given row or column of estimated color channel data. This squared 
difference can, in turn, provide a confidence measure of the sum of the given row or column of color 
channel data, Z x t • Thus, S D implicitly provides a confidence measure of the sum of the given row or 



29. 



The following is in regard to claim 8 and 18. According to Kirk, after deriving the linear functions 
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column of color channel data. Taking all of this into account, it is clear that Kirk teaches (albeit implicitly) 
the determination of one of a confidence and a variance measure (i.e. a confidence measure) for the sums of 
a row or column of color channel data. Therefore, the teachings of Adams and Kirk, when combined in the 
manner described above, address the limitations of claim 8. This same line of reasoning can be applied to 
the rejection of claim 18. 

30. The following is in regard to claims 9 and 10. As mentioned above, the averages, L , I , and 

Q are computed for each of the regions composing the given image. Again, averages are weighted sums. 
These weighted sums are typically computed sequentially - either along rows or columns of the region of 
interest. Thus, the teachings of Adams and Kirk, when combined in the manner described above, address 
the limitations of claims 9 and 10. 

31. The following is in regard to claims 13, 15-17, and 19-20. Note that claims 13, 15-17, and 19-20 
merely claim the method that is implemented by the apparatus of claims 3, 5-7, and 9-10, respectively. 

32. With regard to claim 13, arguments analogous to those presented for 3 are applicable. See the 
appropriate paragraphs above. 

33. With regard to claim 15, arguments analogous to those presented for 5 are applicable. See the 
appropriate paragraphs above. 

34. With regard to claim 16, arguments analogous to those presented for 6 are applicable. See the 
appropriate paragraphs above. 

35. With regard to claim 17, arguments analogous to those presented for 7 are applicable. See the 
appropriate paragraphs above. 

36. With regard to claim 19-20, arguments analogous to those presented for 9-10 are applicable. See 
the appropriate paragraphs above. 

37. Claims 4 and 14 are rejected under 35 U.S.C. 103(a) as being unpatentable over Adams, in view of 
Kirk, in further view of Weisstein ("CRC Concise Encyclopedia of Mathematics", 1999). 

38. The following is in regard to claim 4. As mentioned above, in Kirk's invention, a linear 
relationship is determined, via linear regression, between luminance and chromaticity leading to the 
generation of just two regression coefficients for each chromatic signal. See the equations in column 4 of 
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Kirk. Furthermore, Kirk suggests the usage of "least squares" (Kirk, column 3, lines 27-29) to derive a 
linear relation between each of the chromaticity signals and the luminance signal. Least squares is a well- 
known method of linear regression. Although not explicitly given in Kirk, the formulas for the regression 
coefficients, C 0 i, C n , C 0 q, and C IQ , of eq.l and eq.2 can assume the standard expressions, in terms of the 
corresponding means and variances, given by the least squares method. These are shown in equations (40) 
and (41) on page 1047 of Weisstein. Accordingly, 

_ cov(^,L) 
Cl * ~ a l (32.1) 

X 

C o z =Vz-C lx Mz (32.2) 
where % = I for eq. 1 and % = Q for eq.2, and //^ and ju - are the means of L and %, respectively. Note that 
(32.1) can be expressed as 

where p xL e [-1,1] is the correlation coefficient of % and L . Further note that for linearly related random 
variables % and L , p xL = ±1. From Kirk (see Fig. 2) one can infer that the relation between x and I is 
such that p xL = 1 , in which case, 

C, =tt (32.4) . 

\ a z 

It should then be clear from (32.2) and (321.4) that C 0x and C, x (x = !> Q) of eq.l and eq.2 of Kirk, when 
obtained by least squares linear regression, are of identical form to the coefficients a c and b c ' (C = R, B) of 
claim 4. In particular, since the luminance ( L ) corresponds to the green color channel (G) and the 
chrominance ( % ) corresponds to the red and blue color channels (C = R, B) (see Adams, page 146, Section 
3.4), the equations above are of identical form to those of claim 4, both mathematically and in terms of 
their purpose with respect to a CFA image reconstruction apparatus, obtained either via the combined 
teachings of Adams and Kirk or as claimed by the applicant. 
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32. The usage of least squares to derive a linear relation between each of the chrominance components 
and the luminance would be natural for one of ordinary skill in the art, since it is the simplest and most 
commonly applied form of linear regression (Weisstein, page 1045, paragraph 3). Given this and relative 
ease with which it can be incorporated into an image reconstruction apparatus (Kirk's invention already 
accommodates linear regression), it would have been obvious to one of ordinary skill in the art, at the time 
of the applicant's claimed invention, to incorporate the teaching of Weisstein, with regard to least squares 
fitting, into a system obtained by combining the teachings of Adams and Kirk, in the manner detailed 
above. By doing so, one would obtain CFA image reconstruction apparatus having an image reconstruction 
module that computes the coefficients, of identical form and purpose to those of claim 4. 

33. The following is in regard to claims 14. Note that claim 14 merely claims the method that is 
implemented by the apparatus of claims 4. Therefore, with regard to claim 14, arguments analogous to 
those presented for 4 are applicable. See the appropriate paragraphs above. 



Any inquiry concerning this communication or earlier communications from the examiner should 
be directed to Kevin Siangchin whose telephone number is (703)308-6604. The examiner can normally be 
reached on 9:00am - 5:30pm, Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Amelia Au can be reached on (703)308-6604. The fax phone number for the organization where this 
application or proceeding is assigned is (703)972-9306. 

Any inquiry of a general nature or relating to the status of this application or proceeding should be 
directed to the receptionist whose telephone number is (703)306-0377. 

Kevin Siangchin 
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